Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#2111 - Disable telemetry in DD agent and application #2187

Merged
merged 4 commits into from
Dec 18, 2024

Conversation

coreycarvalho
Copy link

@coreycarvalho coreycarvalho commented Dec 17, 2024

Description

This PR sets DD_INSTRUMENTATION_TELEMETRY_ENABLED in our app and DD_APM_TELEMETRY_ENABLED in the DD agent to false. According to Datadog support, and their documentation, those two env vars are set to true by default, but the federal govt datadog URL does not support those features.

When those are set to true, our DD agent attempts to make calls to the telemetry endpoint, which doesn't exist, causing 404 throttling errors in our logs.

For reference: https://docs.datadoghq.com/tracing/configure_data_security/?tab=environmentvariables&site=gov#telemetry-collection

image

issue #2111

How Has This Been Tested?

Successful perf deploy here: https://github.com/department-of-veterans-affairs/notification-api/actions/runs/12399618902

I deployed this change to perf and those throttling error logs lines are now gone.

image

The deploy finished at around 2:15pm ET and those log items stopped showing up at around 2:19pm ET.

Checklist

  • I have assigned myself to this PR
  • PR has an appropriate title: #9999 - What the thing does
  • PR has a detailed description, including links to specific documentation
  • I have added the appropriate labels to the PR.
  • I did not remove any parts of the template, such as checkboxes even if they are not used
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to any documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works. Testing guidelines
  • I have ensured the latest main is merged into my branch and all checks are green prior to review
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules
  • The ticket was moved into the DEV test column when I began testing this change

@coreycarvalho coreycarvalho marked this pull request as ready for review December 18, 2024 19:29
@coreycarvalho coreycarvalho requested a review from a team as a code owner December 18, 2024 19:29
@coreycarvalho coreycarvalho changed the title Change agent log level to debug #2111 - Disable telemetry in DD agent and application Dec 18, 2024
@coreycarvalho coreycarvalho self-assigned this Dec 18, 2024
@EvanParish EvanParish merged commit 2d340c7 into main Dec 18, 2024
5 checks passed
@EvanParish EvanParish deleted the 2111-dd-logs-throttling branch December 18, 2024 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants